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IDENTIFIERS 
ABSTRACT 

This report presents the philosophy of the Center 
for the study of Evaluation at UCLA. It points out that any 
evaluation of higher education in America must deal with 3 
fundamental problems: 1) diversity of objectives and functions across 
the full range of institutions of higher education; 2) differences 
between college environments; and 3) range of student abilities. A 
more subtle obstacle to effective evaluation has been the traditional 
concept of what evaluation actually is. Traditionally, institutional 
goals ^3^ve been the object of evaluation, but it is more important to 
look at the consequences of particular politics or programs. Although 
"control” and "focus” have characterized the style of inquiry in the 
past, the investigator needs to be adventurous and explorative in 
order to evaluate changing social and educational developments. 
’Exploration” involves searching, probing, and testing alternatives 
and interactions, but it also connotes a freedom to seek new methods 
and values ingenuity. A new model for evaluation programs sees the 
role of xhe evaluator as a social scientist rather than teacher, 
missionary, or reformer. Given this concept of evaluation, it is 
necessary to think about the content of evaluation in a broadly 
inclusive rather than restrictive fashion. The remainder of the 

report consists of a description of the methodological approaches of 
the Center. (DS) 
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PREFACE 



The main purpose of the present report is to describe the 
scope of an evaluation of higher education that is currently 
under way in the Center for the Study of Evaluation. By 
way of introducing the content of this study, we also have 
presented some thoughts about the concept and practice of 
evaluation in education. 

We make no claim to having solved any problems. Instead, 
we have sought, by the very scope of our study, to create 
problems of methodology and analysis which we and others 
must try to solve by dealing with the complexities that a 
comprehensive appraisal requires. We do not discuss these 
technical problems in the present report . 

It should be clear to the reader that there are many as- 
pects of higher education not considered in the studies 
we describe — for example, economic factors, administrative 
^d organizational factors, classroom teaching, political 
influences, etc. We are very interested in these aspects, 
and we hope to consider them more explicitly in future 
studies. For the present, we have tried mainly to extend 
the range of criteria used in judging the impact of educa- 
tion on the behavior of students and former students and 
to extend the range of environmental conditions that may 
have a bearing on this impact . Many commentators have 
said that the most obvious characteristic of higher educa- 
tion in this country is diversity. If one grants some 
truth to this observation, then in his efforts to evaluate 
higher education, he must take this diversity into account — 
in the range of objectives one considers and in the variety 
of institutional settings, one studies . 

In the total program of the Center for the Study of Evalua- 
tion, a considerable range of evaluation problems is being 
studied — classroom learning, elementary school system, pro- 
gram budgeting and cost-benefit analysis, math curricula, 
and other topics. The variety of projects testifies to 
our belief that evaluation is a pervasive concern. The 
higher education project illustrates one type of problem 
and one set of views about it. 
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AN EVALUATION OF HIGHER EDUCATION: PLANS AND PERSPECTIVES 



Most evaluative studies of higher 
education have dealt with one or a limi- 
ted number of aspects of higher education 
rather than with the total enterprise. 

To consider the total enterprise as the 
object of study may well be beyond the 
capacity and ingenuity of any group of 
investigators . Yet the more nearly one 
can approach such comprehensiveness , the 
more likely it is that his results will 
contribute to better judgments and per- 
spectives . 

With some utility, economists charac- 
terize the economic State of the nation 
and predict the probable consequences of 
such policies as lower taxes or higher 
interest rates. Anthropologists charac- 
terize whole cultures. Ecologists are 
increasingly concerned with trying to 
understand man’s total habitat and the 
intricate balance between organisms and 
environments.. Architects and urban plan- 
ners are increasingly concerned with try- 
ing to develop total environments that 
will enhance the quality of life. In each 
case, the efforts have led to a new appre- 
ciation of the complexities and the inter- 
acting forces that must be explored — 
whether of the economy, the culture, the 
biological habitat, or the urban environ- 
ment. 

In evaluations of higher education 
programs, such complexity has been fotmd 
in occasional studies of single institu- 
tions, but studies that encompass large 
number of institutions have typically 
dealt with a smaller number of variables. 
The purpose of some of the larger nation- 
wide studies has been to describe a com- 
plex phenomenon and explain how it works — 
as in the selective distribution of stu- 
dents to colleges or occupations. In 



other studies the purpose has been to 
evaluate how well one type of objective 
is attained — such as changes, in scores 
on personality tests, in scores on 
achievement tests, or in students* plans 
to attend graduate school — with efforts 
to explain what accounts for the perform- 
ance. If one agrees that the training of 
scientific and professional personnel, 
the encouragement of high school students 
to attend college and college students to 
attend graduate school, the acquisition 
of knowledge in broad fields, and the de- 
velopment of personality are in certain 
ways all among the important goals and 
functions of higher education, then these 
inquiries, taken together, have encompass- 
ed a significant range of issues and. out- 
comes. The serious questions to which 
educational evaluators have addressed 
themselves provide clear evidence that 
there is a growing awareness of complex- 
ity and a growing desire for more ade- 
quate evaluations. Examples of signifi- 
cant local and national studies are 
Reeves, 1933; Eckert, 1943; Davis, 1965; 
Thistlethwaite, 1968; Astin, 1968; Trent 
§ Medsker, 1968; and Wolfle, 1954. 

A comprehensive national evaluation 
of higher education must deal with at 
least three fundamental problems: first, 
the diversity of objectives and func- 
tions across the full range of higher 
institutions, necessitating a set of 
criterion or outcome measures that rea- 
sonably reflect and acknowledge 
this diversity; second, the differences 
between college environments, necessita- 
ting the use of suitable, measures for 
describing and classifying some of the 
major, ways in which educational treatments 
and experiences differ from one another; 
and third, the range of student abilities. 



backgrounds, interests, and aspirations, 
necessitating the use of test data and 
other information to identify what kinds 
of students experience what kinds of 
treatments and attain what kinds of re- 
sults . 

There are many published tests 
that could be used in such a comprehen- 
sive appraisal; but using enough of 
them to cover a broad range of criteria, 
contexts, and characteristics has not 
been feasible because such a practice 
would be too time-consuming and would 
require tinusual administrative and fi- 
nancial resources. The obstacle of 
feasibility can be overcome by develop- 
ing shorter tests and by taking advan- 
tage of information that already exists 
in published sources and from previous 
or concurrent studies. The evaluation 
program described later in this report 
represents one serious effort to de- 
velop feasible measures that will 
permit a more comprehensive evaluation. 

A subtler obstacle has been the 
traditional concept of what evaluation 
is. In the field of education, evalua- 
tion has been typically regarded as 
being concerned with how well the spe- 
cifically defined objectives of a pro- 
gram are attained or, in experimental 
designs, with determining whether or 
not one educational treatment produces 
results different from those of an al- 
ternative treatment or a control group. 
The requirements for such a model are 
rigorous: (a) clear objectives — behav- 

ioral ly defined, (b) random assignment 
of subjects to different treatments, 

(c) clearly differentiated treatments, 
and (d) criterion measures explicitly 
related to the objectives. There is 
some recognized slippage in the model. 



however, for Investigators are usually 
admonished to be on the lookout for 
"side effects" and for alternative ex- 
planations of their results. Neverthe- 
less, the model has served education 
well and has led to many solidly docu- 
mented conclusions, as a reading of the 
Handbook of Research on Teaching (Gage , 

1963) will confirm. The shortcomings 
of the model can be illustrated by some 
rhetorical questions: What does one do 

when not all the relevant objectives are 
manifested in directly observable speci- 
fic individual behavior? What does one 
do about deliberately trying to measure 
effects that are not objectives of the 
program? What does one do when random 
assignment of subjects to treatments 
cannot be accomplished? What does one do 
when he lacks clearly differentiated 
treatments? Or when a treatment changes 
during the course of the study? The tra- 
ditional evaluation model provides no 
answers to such questions. Perhaps the 
answer it does provide is illustrated by 
^ New Yorker cartoon that appeared when 
the automatic record changer first came 
on the market— showing records flying 
off in all directions with the caption "If 
it isn*t Bach, the hell with it I" 

One of the most hopeful developments 
in higher education today is the willing- 
ness of many colleges to- try new programs 
or, in the image of the cartoon, to play 
new music. The book Higher Education : 

Some Newer Developments (Baskin, 1965) siq)- 
ports this fact. There are periods of so- 
cial change that correctly can be described 
as turbulent. We are living in one of them. 
One cannot make change stand still so that 
an evaluator can compare "experimental ys^ 
control" treatments. If there is a course 
in Negro History, one cannot exclude stu- 
dents who want to take it. If new math 
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is a good thing, one cannot prevent stu- 
dents from encountering it. Indeed, the 
whole notion of random assignment of sub- 
jects to treatments is usually not toler- 
ated in the reality of society where the 
pressure is for equal opportunity and 
the best possible treatment for everyone. 
Such controlled conditions can be set up 
only in a laboratoiy that is removed from 
the moral and dynamic imperatives of the 
larger society, and the results of such 
laboratory experiments may or may not be 
reproducible in more realistic settings. 

If educational evaluators are to deal 
effectively with large problems that can- 
not be simulated in a laboratory- -problems 
such as the effectiveness of a total insti- 
tution, of a class of institutions, or of 
higher education in the United States — 
then they need a concept of evaluation that 
is both more comprehensive and more flex- 
ible than the familiar experimental model. 
For this level of complexity and reality, 
the evaluator must ask different ques- 
tions, proceed in a different style, and 
have a new view of his role and purpose. 

The central question is not ’’What 
are the objectives?”. The central ques- 
tion is ”What are the consequences?” If 
one should ask himself the question, 

’’What are the objectives of the United 
States in Vietnam?” , no doubt some an- 
swers would come to mind. But if instead 
one asked himself the question, ’’What are 
the consequences of the war in Vietnam?”, 
a much greater range of inquiry would im- 
mediately be suggested and required. An 
analogous situation exists with any large 
scale social phenomenon, and higher edu- 
cation in the United States is one such 
phenomenon. What are the consequences of 
the existence of higher education--of the 
' fact that there are some 2,400 colleges, 
universities, and junior colleges, that 
there are some 6,000,000 or more enrolled 



degree candidate students, and that more 
than half of the nation’s high school 
graduates enter an institution of higher 
education? What if there were no higher 
education institutions? What then would 
be the consequences? It seems obvious 
that the range of one’s inquiry is 
guided by the questions one asks. ’’Ob- 
jectives” are a subheading under ’’conse- 
quences.” Of course, one hopes that 
among the consequences are some which 
are intended as objectives, but looking 
at the extent to which objectives are 
achieved will not answer the larger 
question about consequences. Therefore, 
the first requirement for a new model 
of evaluation is to begin with the ques- 
tion ’’What are the consequences?”. 

The second necessary element in a 
new model is one which relates to the appro- 
priate style of inquiry. An apt term for 
this is ’’exploration.” Traditionally, 
the style of inquiry has been character- 
ized by the words ’’control” and ’’focus.” 
Exploration is a freer style— one, which 
encourages hunches, is imcommitted, and 
seeks discovery. If the program one 
hopes to evaluate is continually changing 
in methods, materials, personnel, and 
subjects, it is not necessarily true that 
it cannot be evaluated. To the contrary, 
one may discover that programs which are 
being modified continually are more 
effective than programs that remain rel- 
atively static. The spirit of the evalu- 
ator should be adventurous. If only that 
which could be controlled or focused were 
evaluated, .thbn^^a great many important 
educational and social developments would 
never be evaluated--at least not by 
’’evaluators;” that would be a pity. To 
suggest that the style of the controlled 
experiment needs to be replaced by an 
exploratory style does not mean that one’s 
approach should be any less scientific. Ex- 
ploration involves searching, probing and 
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testing alternatives and interactions. It 
can be tough -minded, rigorous, and theory- 
based. But the word exploration also con- 
notes a freedom to look around, to seek 
new measures and methods, and to value in- 
genuity and curiosity. 

The third element for a revised con- 
cept of evaluation is a new view of the 
role of the evaluator and the purpose of 
evaluation. Historically, many people 
have seen evaluation as an instrument of 
reform. The reason for evaluating any pre- 
sent activity or program was to improve 
it. As a result of this approach, the 
parties to the activity or program had to 
be involved in its evaluation because 
their very involvement would increase the 
likelihood that they would be willing to 
change in the light of the findings. Thus, 
the process of carrying out an evaluation- 
group participation and cooperation — was 
directly related to achieving the purpose 
of evaluation, namely, change and improve- 
ment. Implicitly, the role of the evalua- 
tor was in some respects the role of mis- 
sionary and reformer. Another common view 
of the purpose of evaluation is that it 
exists to give feedback to the decision- 
makers (presumably teachers and adminis- 
trators) . In this case the role of the 
evaluator is a kind of staff officer to the 
practitioner. Neither of these views seems 
c^uite appropriate for evaluations that deal 
with large, complex social or educational 
programs such as higher education in the 
United States. A clearly identifiable 
number of individuals who can be called 
the decision-makers does not exist. In 
realpolitik reforms are brought about by 
persuasion, pressure, consensus, and con- 
viction — not by evaluators. VIhat evalua- 
tors produce may, of course, contribute 
to the decisions that are made. One 
hopes that they will; but whether or not 
evaluators contribute to such decisions 



is not the issue. Whether or not the 
evaluator is also the politican and 
thus carries out his evaluation in 
ways designed to generate certain con- 
victions the issue. A more appro- 
priate concept of evaluation would 
define the role of the evaluator as 
that of a social scientist and the pur- 
pose of evaluation as that of providing 
more complex bases for informed judgment. 
When asked about the relation between 
social scientists and administrators, 
Harold Lasswell is reported to have said 
that the role of the social scientist is 
to complicate the task of the decision- 
makers. The evaluator, as a social scien- 
tist, should see this pursuit as his role, 
too. 

A concept of evaluation, appropriate 
for the study of large and complex insti- 
tutions, can be summarized briefly as 
follows: 

1. It begins with the central ques- 
tion, "What are the consequences?" rat- 
her than with the more limiting question, 
"What are the objectives?" 

2. Its style of inquiry is more 
aptly characterized by the word "explo- 
ration" than by the words "control" 
and "focus." 

3. It sees the role of the evalua- 
tor as a social scientist rather than 
as a teacher, missionary, reformer, or 
staff officer to the practitioners. 

4. Its purpose is to provide more 
complex bases for informed judgment. 

Given this concept of evaluation, one 
should be persuaded to think about the 
content of evaluation in a broadly in- 
clusive rather than restrictive fashion. 




In June, 1966, the United States 
Office of Education established at UCLA 
a Research and Development Center for 
the Study of Evaluation. The Center has 
been concerned with the theory and prac- 
tice of evaluation at all levels of edu- 
cation --elementary, secondary, and higher-* 
and with programs at different levels of 
complexity— from classroom learning to 
large school systems. One of the activ- 
ities of the Center is focused on evalua- 
tion in higher education. A description 
of this activity constitutes the remain- 
der of this report. The long-term pro- 
gram is more ambitious than what we shall 
describe here, for its ultimate intent 
is to explore a broad range of social 
indicators and consequences of higher 
education- -both positive and negative--of 
the sort suggested earlier in this report. 
But our exploration of such widely-ranged 
consequences has not yet progressed to 
the point where we feel that some publi- 
cation is merited. In this report we 
shall describe one comprehensive study 
that is now under way— a study concerned 
primarly with a range of outcomes of 
higher education that may be seen in the 
behavior of students and adults exposed 
to the college experience, and with the 
institutional characteristics and indivi- 
dual experiences that may explain the 
extent to which different outcomes are 
achieved. 

Growth in knowledge and understand- 
ing, intellectual skills and interests, 
heightened awareness, appreciation, 
values, attitudes, citizenship, moral 
sensitivity, and activities as producers 
and consumers of ’’the good life” — all 
these and more are commonly discussed 
among the objectives of higher education. 
Yet such a broad range of goals is sel- 
dom dealt with in systematic evalua- 
tions of higher education. More fre- 
quently, judgments about the quality of 



institutions are based on only one 
type of criterion--their assumed^ 
or measured intellectuality. This 
method of judgment may define the 
place of the institution in the sys- 
tem but only on one dimension, and 
it reveals nothing about the effec- 
tiveness of the institution. Many 
colleges and universities, which 
are necessarily low on an intellec- 
tuality dimension because of the 
nature of their student input, may 
be highly effective in other respects 
such as the amount of change produced 
in their students or the role of their 
graduates in community service and 
active citizenship. As we approach 
more or less universal higher educa- 
tion, evaluations based on a single 
criterion are not only increasingly 
inadequate, but they may lead to a 
distorted national perspective of the 
different roles and differential ef- 
fectiveness of higher institutions. 

Moreover, most evaluations have 
focused on individual measurement 
rather than on institutional meas- 
urement. Yet higher education in any 
organized sense takes place in some 
sort of institutional setting. Re- 
cent research (Pace, 1967) has clearly 
demonstrated that large differences 
exist among different institutional 
settings. With more than half of all 
the high school graduates in the coun- 
try now going to college and distribut- 
ing themselves among institutions which 
in many cases differ radically from one 
another, how much do we really know 
about the kind of impact these varied 
institutions have upon their students? 

In any national appraisal of higher 
education, there are really three poten- 
tial sources of distortion i first, an 
inadequate range of criterion variables; 
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second an inadequate range of contextual 
or environmental variables; and third, 
an inadequately representative population 
of individuals and institutions. In the 
studies we are conducting, we are attempt- 
ing to minimize each of these potential 
sources of distortion. 

First of all, our population includes 
approximately 100 colleges and universi- 
ties from all parts of the coimtry, large 
and small, public and private. Within 
this national assortment of educational 
institutions we have included selective 
examples of each of eight major cate- 
gorids, or types, as follows: 

1. Highly selective liberal arts 
colleges, private, nonsectarian 

2. Strongly denominational liberal 
arts colleges 

3. General liberal arts colleges 

t 

4. Highly selective universities, 
public and private 

5. General comprehensive universi- 
ties, public and private 

6. State colleges and oth’er uni- 
versities having less extensive 
graduate programs tha.n general 
comprehensive universities . 

7. Colleges having a mrjor emphasis 
on teacher education 

8,. Colleges having a major emphasis 
on engineering and sciences 

Within each of these colleges and univer- 
sities, we are obtaining data from random 
samples, of incoming freshman, upperclass- 
men", aiid aliamnii with larger samples from 
the. larger, institutions and smaller sam- 
ples from the s, mailer institutions. 



Aliamni and upperclassracn are being 
tested in the winter and spring of 1969 
and freshmen in the fall of 1969. 

The basic content of the question- 
naire developed for this evaluation 
study is the same for each of the three 
groups, except for minor changes in 
wording and a few additions or omis- 
sions of items that would be applicable 
to only one of the three gro^s. Our 
description of the content will refer 
to the alumni questionnaire . 

The first section of the question- 
naire consists of activity scales deal- 
ing with a broad range of involvement 
in. contemporary society, and culture. 

Each of the activity scales contains 
from nine to twelve items. The topics 
of these scales are as follows ; com- 
munity affairs, national and state 
politics, art, music, literature, drama, 
education, science, religion,, int er-cul.- 
tural affairs, and international affairs. 
The internal structure of each, of the 
scales is the same in. that each scale 
includes activities ranging from. some. . 
which are relatively simple, commonplace, 
and easy to do. to ones, which, involve, in- 
creasing amounts of interest, time, and 
commitment. The instructions for each 
scale are to "check each, statement that 
describes an activity you have engaged 
in during the past year.". The following 
statements describe the scale for art 
activities : 

During the past year; 

I talked about ait with my 

i” friends. 

I read critiques or reviews of art 
shows or exhibits in the newspapers 
or magazines. 

I visited an art gallery or art 

museum. 



o 

ERIC 



I attended an exhibition of contem- 
porary painting or sculpture. 

I read one or more books about art, 
artists, or art history. 

I bought a painting or piece of 
sculpture . 

I attended an art study group or 
workshop. 

I contributed money or time in sup- 
port of some activity related to 
art. 

I did some creative painting or 
other art work myself. 

The number of activities checked in 
each scale provides a measure of the 
amount and depth of one*s participation 
and interest. The number of different 
scales in which, one checks more than 
some minimal number of statements, pro- 
vides a measure of the breadth of one’s 
interests and involvement. Moreover, 
since many of the scales have certain 
roughly parallel items, one can derive 
additional, indexes that have some. rele-. 
vance to the consequences or effects of 
higher educati n. For example, one can 
obtain a score indicating the number of 
different fields or topics about which 
one talks or discusses with his friends, 
an index of the momber of different 
fields in. which one has read a book, and 
the number of different fields in which 
one attempts to keep, up with current 
events through the newspapers , magazines , 
or television. With the arts (3-rt, music, 
literature, and drama) three additional 
indexes can be derived: an index of 

the extent to which one exposes himself 
to a variety of opinions through the 
reading of reviews and criticisms, an 
index of one's exposure to contemporary 
works , and an index of engagement in 
personally self-expressive behavior. 

From the civic and political scales, one 



can obtain an index of ‘activism. Thus, 
the activity scales provide a rich source 
of information relevant to many of the 
intended or potential outcomes of higher 
education. Moreover, some of the items 
in these scales are similar or identical 
to ones which were initially developed 
by the writer for alumni surveys at Ameri- 
can University and Syracuse University 
in 1946-47 and which were also used in a 
nationwide survey of college graduates 
made in 1947-48 by the rese^ch division 
of Time magazine. The possibility of ob- 
serving 20 -year trends is an added bene- 
fit of significance. 

The second section of the question- 
naire is designed to provide two kinds of 
measures: first, a measure of knowledge 

and awareness about certain major changes 
that are taking place in American society 
and, second, a measure of attitudes toward 
such changes. Knowledge of what is and 
is not happening in society is fundamental 
to any intelligent adaptation to changing 
conditions and in this sense may be re- 
garded as a culminating objective of lib- 
eral education. The primary sources for 
constructing items about major social 
changes have been various publications 
from the American Academy of Arts and 
Sciences, the National Research Council, 
and other governmental or independent 
bodies of social, analysts . In range of 
content, the various items deal with 
changes in the labor market, the economy, 
the environment, education, science, gov- 
ernment, industry, snd other important 
aspects of the American scene. Here 
are a few sample statements in the test; 

More people are coming to realize 
and accept the value of self-ex- 
pression--for example, in the arts. 
Scientists and professionals are 
having an increasingly important 
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influence on economic and govern- 
mental policies. 

Increasingly, government is con- 
trolling the markets for the most 
advanced industries. 

To each of these statements and some 
25 others like them, two types of re- 
sponses are solicited! first, a know- 
ledge response indicating whether one 
thinks that the statement is or is not 
generally true, and second, an attitude 
response indicating whether one thinks 
the change described by the statement 
would be desirable or undesirable if 
it has in fact occurred or is occurring. 
The recognition of change and the readi- 
ness to deal with it can thus be compared 
with the denial of change and resistance 
to it. 

.The third section lists various 
educational objectives or potential' bene- 
fits and asks the respondents to rate 
the extent to which the college experi- 
ence was influential in relation to 
those objectives or benefits. In an- 
other section, we ask several questions 
about the existential value of college 
experience, apart from any instrumental 
or vocational benefits. 

The sections of the questionnaire 
which we have described are concerned 
with criterion measures, that is, with 
activities and interests, knowledge, 
attitudes, judgments, and feelings re- 
lated to a broad range of intended or 
relevant outcomes of higher education. 
The last two sections of the question- 
naire deal with the nature of the school 
nd college experience and with a va- 
riety of personal information that is 
intended to throw light on some of the 
individual and environmental conditions 
that may help to account for performance 
on the various criterion measures. 



With respect to school and college 
experience, we have included items 
about major field, academic perfor- 
mance, participation in extra-curricular 
activities, characteristics of the col- 
lege environment, the continuity of 
college experience, the extent of formal 
education beyond college, aspects of the 
college experience that stand out in mem- 
ory, and some corresponding information 
about the type of high school attended, 
high school achievement, and participa- 
tion in various high school extra-curri- 
cular activities. 

The section dealing with personal 
information includes the usual kinds of 
census data such as age, sex, marital 
status, nxjmber of children, race, eth- 
nic background, and occupation. It also 
includes questions about personal and 
family background — economic, cultural, 
political, and religious. In addition, 
there are items inquiring about the var- 
ious parts of the country in which the 
respondents have lived both before and 
after college and about the extent to 
which they have traveled in other parts 
of the world, both before and after col- 
lege. Finally, in the last part of this 
section, there is a brief personality 
test and a brief vocabulary test. The 
personality test items are related to 
three characteristics: (a) theoretical 

orientation, (b) complexity, and (c) 
autonomy. The vocabulary test calls 
for definitions of words of various 
levels of difficulty. 

In summary, the total questionnaire 
with minor modifications, is being ad- 
ministered to samples of incoming fresh- 
men, upperclassmen, and alumni in each 
of approximately 100 colleges and uni- 
versities across the country. It in- 
cludes a broad range of criterion 
measures, information about the college 



experience, and data about the background 
and personal characteristics of the re- 
spondents. 

Since one of our purposes in this 
evaluation is to assess institutional 
effectiveness, we are also obtaining 
much more data about institutional char- 
acteristics than will be obtained from 
the basic questionnaires. Some of this 
information comes from studies that have 
already been made; some of it comes from 
directories and other sources of demo- 
graphic information, and some of it will 
come from instruments which we are still 
in the process of developing or pretest- 
ing. For example, for most of the insti- 
tutions included in cur sample, we have 
data about the' institutional environment 
obtained from the previous administra- 
tion of College and University Environ- 
mental Scales and from scores obtained 
by the Environmental Assessment Techni- 
que reported in the book, Vlho Goes Where 
to College? (Astin, 1965). From direc- 
tories we are recording a variety of 
demographic information about the insti- 
tution such as size, complexity, control, 
characteristics of faculty, curriculum, 
and the involvement of the institution 
in research. These are potentially im- 
portant aspects of the environment in , 
which learning and development presum- 
ably occur. 

Beyond these previous and published 
sources, we are also developing a number 
of new ways for characterizing the edu- 
cational environment. At the present 
time, we can describe six such measures. 
They are in various stages of develop- 
ment . 

The first two are already completed, 
and they consist of scoring selected 
items that are included in College and 
University Environment Scales . One 



consists of 22 items which, when com- 
bined provide an index of Campus Morale. 

The other consists of 11 items from CUES 
which, when added together, provide an 
index called Quality of Teaching and 
Faculty-Student Relations . 

The next two contextual or environ- 
mental variables are in the pretesting - 
stage. One of these is concerned with 
the style of learning or academic ef- 
fort that is characteristic of a campus. 

Last year we had a small panel of stu- 
dents from each of four quite different 
institutions keep a detailed time- log 
of academic activities for a consecu- 
tive seven-day period. In analyzing 
these time-logs, a number of institu- 
tional differences were apparent. In 
some institutions, for ex^ple, there 
were numerous entries indica. ing con- 
versations between students and fac- 
ulty members outside class. At other 
institutions, there were very few such 
entries . In some institutions , there 
were several entries describing con- 
versations with other students about 
academic matters; whereas, in other 
institutions, there were very few such 
entries . Although* there were no great 
differences between institutions in 
the total amount of time which students 
reported spending on academic activities, 
there were noticeable differences in 
the concentration and intensity of that 
time— differences, for example, in the 
frequency of uninterrupted periods of 
study far into the night and in the 
amount of study that occurred on week-ends. 
There were further institutional differ- 
ences in such activities as the amount of 
conversation in class, the amou’^t of writ- 
ing done, and the amount of academically 
related but non-assigned reading done. 
These and other items are providing a 
basis for constructing a brief scale which 
will identify major differences among'the 
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learning styles that characterize each 
institution. 

The other contextual variable that 
is currently in the* pretesting stage is 
concerned with identifying the institu- 
tional stance toward student freedoms 
and discipline— a stance which ranges 
from protective and ptinitive to devel- 
opmental and permissive. . 

Two other contextual variables on 
which we are working are currently in 
the developmental stage. For one of 
these , we are trying to devi se a way 
of characterizing the peer group rela- 
tions and influences that appear to be 
dominant on the campus. For the other, 
we are attempting to devise some meas- 
ure of institutional anonymity and 
alienation. 

As we continue to think about the 
institutional environment, we shall 
probably develop still further ways of 
characterizing it. The point is that 
much of what has been done in the past 
hais been concerned with relatively 
small aspects of the environment and 
educational experience. In the long 
run, whether one course is taught b*' 
lecture or discussion or some comb-.. 
nation of the two is probably ins‘ 
f icant . Whether some of the coui. 
make extensive use of television 
others do not is probably equally ^ - 
insignificant, as is the question 
whether the general education re- 
quirements must be met by taking a 
half-dozen specifically designated 
courses or whether they can be met by 
selecting a pattern of courses among 
a larger number of alternatives . It 
may, on the other hand, be important 
to know the extent to which students 
encounter variety rather than same- 
ness in the teaching style of the 



faculty, and it may be important to know 
whether the curriculum and other academic 
regulations are characterized by flexi- 
bility or rigidity. What we are looking 
for are pervasive and stylistic differ- 
ences that can be felt throughout the col- 
lege experience rather than particular 
differences that are evident in only one 
or a few segments of the college environ- 
ment. 

In the analysis of results from the 
national field study, we shall be seek- 
ing answers to a fundamental but complex 
question: what personal and backgroimd 

characteristics, what school and college 
experiences, and what characteristics of 
institutional environments are associa- 
ted with each of some 50 or more differ- 
ent and relevant criterion measures? 

The study we have described differs from 
previous national studies in two very 
important ways: first, it includes many, 
rather than few criterion measures; and 
second, it includes new kinds of poten- 
pervasive environmental or con-^ 
textual measures. The range of criterion 
measure is really the crucial difference 
between this and previous studies. It 
is one thing to identify personal dis- 
positions, educational experiences, and 
environmental conditions that are pre- 
dictive of one criterion-such as con- 
tinuing one’s education in graduate 
school; but it is quite another thing 
to ask; and discover whether or rot pre- 
cisely these dispositions, experiences, 
and conditions that are positively as- 
sociated with going to ^aduate school 
may be negatively associated with com- 
munity service, political involvement, 
or interests in art, music, literature, 
and drama. It is at least conceivable 
that the conditions conducive to some 
kinds of outcomes may be quite different 
from the conditions conducive to other 
types of outcomes . It is the concern 
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with different outcomes, each of which is 
important and relevant to 'education and 
the larger society, that gives a new 
level of complexity to the study we are 
making. 

The essence of evaluation is to 
place values on, and the important aspect 
of this statement is that the noun values 
is plural — not singular. If one*s crite- 
rion for building a new highway is to 
locate it so that its construction in- 
volves the least amount of earth movement 
and direct engineering cost, one looks 
logically at a variety of factors that 
might be related to this criterion and 
pays attention to those that have the 
highest relationship. Other values, 
such as the displacement of people or 
the effect on ecology, are ignored be- 
cause they are not included among the 
criteria. The result may be a perfectly 
good research study and, at the same 
time, a very bad evaluation. The con- 
cept of evaluation that underlies our 
study requires a consideration of many 
criteria or possible outcomes. If a 
national study of higher education 
does not meet these requirements rea- 
sonably well, it is not and should not 
be interpreted as an evaluation. 

It is our hope that these studies 
will make some contribution not only 
to national perspectives about higher 
education but also to local institu- 
tional self-study and evaluation. The 
various scales and other measures de- 
veloped for. the study can be used 
separately or collectively in further 
studies. Also, since each participa- 
ting college and university will re- 
ceive a report of the results obtained 
from its own students and alumni, local 
committees or other groups can examine 
the data and consider their significance. 
Finally, as we analyze the data from 



all the college^ and universities, we 
hope to uncover connections between 
various practices and outcomes that 
may have influence on the future ef- 
fectiveness of institutions. 
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